Extracting the Lowest Frequency Words: Pitfalls and Possibilities

s or to the complete newspaper corpus. This raises the question of whether better results might have been obtained if the complete data sets had been used. In principle, more data might imply more power. At the same time, more data also entails the risk of more noise. At least for our af data, enlarging the complement leads to worse performance. When we allow any sentence that contains af in our analyses, F decreases from 0.31 to 0.23 for G2. When we base the analyses on the complete newspaper corpus, F reduces further to 0.19. The reason for this decrease in performance is probably due to the W C-ratio being very low for all practical window sizes, i.e., at the very left part of the saw-tooth-shaped pattern characterizing Nsig as a function of W C. Consequently, any low-frequency word is singled out as a signiŽcant item whenever it occurs at least once in the window. Given the ZipŽan structure of word-frequency distributions, a great many spurious low-frequency words are extracted. As mentioned in the introduction, the received wisdom is that the windowing method is unreliable for events with a frequency of less than 5. By means of an analysis of the behavior of statistical tests for 2 2 contingency tables with sparse data, a method for optimizing the use of these tests has been developed. We hope that this technique will prove to be useful for domains in which the extraction of low-probability events is crucial.